Speech recognition systems can have difficulty recognizing proper names, particularly surnames. In the case of mobile telephones, for example, this is addressed by sending a contact list or other information from the mobile telephone to a server performing the speech recognition. For example, a user may speak into a mobile telephone. Along with the resulting speech data, the mobile telephone may send the user's contact list to the speech recognition server. The speech recognition server then uses that contact list to attempt to identify proper names in the speech data. This process, however, can raise privacy concerns.